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ABSTRACT 

In this study, the authors examined the relationship between the various NSSE (National Survey of 
Student Engagement) survey items or classifications and the ETS Major Field Test (MFT) in 
Business. The sample consisted of 87 business students at a small liberal arts college. Many of 
the survey items were linked positively to the ETS Major Field Test, The results suggest that MFT 
is a better measurement than GPA for NSSE. As a noteworthy finding from the analysis, we need 
to mention that MFT is a better measurement of student engagement described in the NSSE rather 
than GPA. Therefore, it seems that the MFT is a valid external measurement of student 
engagement. In particular, the MFT shows a relative strength in assessing “Integration of 
Diversity into Coursework. ” 
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BACKGROUND 


yy 'nstitutions of higher education are struggling to measure and report student outcomes. By around 2011, 
•/ internal and external assessment outcomes are expected to be measured, reported, and available on the 
institution’s website. This ‘culture of evidence’ will be expected of universities with prospective students 
and parents being able to view the data and potentially be able to make a more informed or uniform decision when 
choosing a university by comparing results between institutions for various outcomes. A variety of standardized 
tests such as the ETS Business Major Field Test and NSSE have been used as an external student outcome measure 
that may show the value of the educational experience for the student provided by the institution. 


From an institutional perspective, Mirchandani et al. (2001) speculated that two simple questions needed 
answers “a) What do we want the students to know and/or be able to do and (b) how do we know that they know it 
and/can do it?” (p.51). Common outcomes that institutions want their students to know and/or be able to do include 
critical thinking, problem solving, written and oral communication, global awareness, ethical reasoning, and many 
others. Many of these outcomes are measured with rubrics developed internally by institutions and can demonstrate 
the student’s progress at the university. Other measures of what the students know include the use of standardized 
tests such as the ETS Business Major Field Test and NSSE. 


ETS DISCUSSION 


In addition to the previously mentioned outcomes, business students are also expected to have a knowledge 
base in the business core including accounting, economics, management, quantitative business analysis, finance, 
marketing, legal and social environment, information systems and international issues. The ETS MFT in Business is 
an evidence based system “designed to measure a student's subject knowledge and the ability to apply facts, 
concepts, theories and analytical methods. Some questions are grouped in sets and based on diagrams, charts and 
data tables. The questions represent a wide range of difficulty and cover depth and breadth in assessing students' 
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achievement levels.” (http://www.ets.org/mft/about/content/bachelor_business). ETS provides each institution with 
a report that includes information on individual student scores along with a group mean score and percentiles. A 
review of the literature revealed only three studies regarding the effectiveness of the ETS MFT in Business but the 
articles did reveal interesting results and address some concerns we had regarding the validity of the test. 

For example. Black and Duhon (2003) stated “ETS addresses content validity by using faculty committees 
and expert consultants to construct and evaluate the items on the business field test” (p. 93). Black & Duhon (2003) 
also found men score better on the ETS MFT than women, “a business core GPA that is one point higher is 
associated with an ETS test score that is 7.49 points higher” (p. 94), a one-point higher composite ACT showed an 
increase of 1.51 points on the ETS, age plays a positive factor in ETS scores, and management majors were at a 
disadvantage on the ETS relative to other majors. Many of these same comparisons were made in our study. In 
addition, they used the results for either learning information (program and student assessment) or continuous 
improvement (program development and student development). Two steps they took were to suggest that 
underachieving students “participate in work-study programs to better integrate classroom learning and real-world 
applications” (p. 97) as they saw a positive association to increased ETS scores and exposure to the business 
environment and underachieving students could work with upperclassmen, faculty members or business leaders 
from the community. These are valuable insights into understanding student learning and can help guide course and 
curriculum revisions. 

Another study done by Allen and Bycio (1997) found SAT-V (SAT - Verbal) and GPA-BUS (GPA - 
Business) were related to MFAT-B (Major Field Achievement Test - Business). “The significantly moderately 
large correlation observed between the MFAT-B and GPA-BUS is a positive outcome that supports the criterion- 
related validity of the test” (Allen and Bycio, 1997, p. 511-512) but they also felt “...the results would have been 
more positive had the MFAT-B been more strongly related to performance in the Business Core and less strongly 
related to, for example, SAT scores” (p. 512). This may mean that an input variable such as SAT or even good test 
taking skills can have a significant impact on MFAT-B scores. Regardless of this, their results do provide some 
assurance that the test functions as intended” (p. 514). 

In the study by Mirchandani, Lynch and Hamilton (2001) also recognized the impact of SAT scores but 
found quantitative and qualitative factors to be significant. They believed that concentrating “on the material from 
the courses that compromised the quantitative factor (Calculus, Accounting I & II, Finance, Operations 
Management, and Management Information Systems) because that factor was the most significant process variable, 
although the effect size was much smaller than that of the SAT” (pp. 54-55). 

NSSE DISCUSSION 

Many institutions in the past decade have participated in the NSSE survey to better understand how certain 
practices inside and outside the classroom may lead to increased student learning. According to the NSSE website, 
“NSSE Student engagement represents two critical features of collegiate quality. The first is the amount of time and 
effort students put into their studies and other educationally purposeful activities. The second is how the institution 
deploys its resources and organizes the curriculum and other learning opportunities to get students to participate in 
activities that decades of research studies show are linked to student learning” (http://nsse.iub.edu/html/about.cfm). 

Pascarella, E.T., Seifert, T.A., & Blaich, C. (2010) state “one of the major assumptions of the NSSE is that 
in measuring the extent to which students engage in such practices, one is indirectly measuring student cognitive and 
personal development during college” (p. 18). This study looked at the relationship between NSSE and other direct 
measures of student development and learning (effective reasoning and problem solving, moral character, inclination 
to inquire and lifelong learning, intercultural effectiveness, and personal well-being). A compelling finding of their 
study was that “Across all liberal arts outcomes, the most influential NSSE benchmark appeared to be the Enriching 
Educational Experiences scale, which had significant associations with three of the seven outcomes: effective 
reasoning and problem solving, moral character, and intercultural effectiveness” (p. 20). A 2008 research study 
conducted by Pascarella, et al. confirmed the “primary assumption of NSSE that in measuring student exposure to 
and engagement in empirically-vetted good practices, one is essentially measuring experiences that yield desired 
cognitive and personal development during college. Thus, collecting data on student experiences provides a window 
from which institutions can extrapolate to student learning” (p. 35). 
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Finally, the study conducted by Gordon et al. in 2008 focused on cumulative GPA, first-year retention, job 
attainment upon graduation, and the decision to pursue a graduate degree as indicators of success in their student 
body. Two of their findings were of particular interest to us. First, they found “a stronger relationship for seniors 
than freshman, suggesting that engagement has an influence over the career of the student - something not readily 
seen after a semester and a half of experience” (p. 26). Second, one item on the NSSE provided explanatory power 
in retention, job attainment upon graduation and the decision to pursue further education. This item was the 
participation in some sort of practicum such as clinical work, co-op or internship. Our data supported this research. 

Understanding the concerns regarding the ETS Business MFT and NSSE was an important aspect of 
moving forward with our research. As noted by Mirchandani, Lynch and Hamilton (2001), “GPA has strong 
internal validity and reflects student performance in the context of the curricular and pedagogical priorities at a 
particular institution, whereas the standardized test scores provide greater external validity and allow an institution 
to quickly establish a baseline and benchmark against national norms” (p. 56). We believe that both the ETS 
Business MFT and the NSSE have the necessary external validity for us to utilize them in further studies and 
possibly understand how one might even affect the other. 

METHODOLOGY 

In the pilot study done by Ward et al. (2009), we explored the relationship between NSSE engagement 
items and classifications and ETS Business MFT scores. This research is a continuation of that pilot study. At the 
end of the spring 2010 semester, we had a sample size of 87 students. As stated in the previous paper, the sample 
consists of student who completed both the ETS Business MFT and the abbreviated NSSE survey. 

The response values of 16 survey items directly related to the NSSE have been standardized using the 
standard deviation and the mean value of each survey item. Also, the standardized scores of the survey items are 
grouped based on the four categories of the NSSE survey. The standardized scores of the survey items and the 
categories of the NSSE survey are analyzed using both the Ordinary Least Squares regression and the partial 
correlation coefficients. 

RESULTS 

The four NSSE areas we focused on for the analysis were Level of Academic and Challenge, Active and 
Collaborative Learning, Enriching Educational Experiences and Integration of Diversity into Coursework. As stated 
in our previous paper, we added or modified four questions on our NSSE survey. “We added a question regarding 
the types of co-curricular activities the student had done in the past year and the average number of hours worked 
off-campus in the 2008-2009 academic year. On the question regarding the number of hours per week the student 
participated in co-curricular activities, we modified the question to exclude intercollegiate sports in order to capture 
more volunteer activities (such as joining a club, being a resident assistant, community service, etc.). Finally, we 
asked the students if they had held a position of leadership on campus (Ward et al., 2009). These additions or 
modifications proved to be insightful. 

With our current set of data, we saw positive increases in Co-curricular activity in the following areas: 
Community service and volunteering, community based clubs or groups, creative or performing arts, campus 
publication and fraternity or sorority. Our institution has focused its efforts on getting students engaged in multiple 
activities on campus in an effort to improve the student’s experience on campus, broaden their horizons, and 
improve retention. The number of hours per week devoted to these activities increased as well with 18.4% of 
students spending 8 or more hours per week on the combined activities. A rather unsettling trend, however, is 
students are also spending additional hours working off campus. While the previous study showed 19.5% of 41 
students worked 21 or more hours off campus, this percentage increased to 26.4% of 87 students. The economic 
downturn is the likely cause of this trend. Finally, holding a position of leadership on campus remained relatively 
stable at 34.5%. This item also had a marginal correlation to MFT of .239 at the .05 level of significance. Due to 
our small campus, there are a significant number of opportunities for students to hold a position of leadership such 
as student government, resident assistant or club officer. 
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Several notable shifts occurred in the descriptive data and are listed in Tables 1 and 2. Table 1 shows the 
GPA in the bottom two and top segments remained relatively stable; there was an upward shift in from 14.6% to 
23% in the 3.00-3.33 range as well as a downward shift from 41.5% to 32.2% in the 3.34-3.66 range. Table 2 shows 
the overall ETS MFT scores trending upwards as the scores between 151-165 and 166-180 revealed higher 
percentages of students falling into those categories. In addition, the partial correlation between MFT and GPA 
increased from .361 to .492 supporting the work done by Carini et al. (2006) that student engagement is often 
positively related to GPA. These will be interesting phenomena to track as we gather more data. 


Table 1. GPA 



N= 41 

N= 87 

2.66 or lower 

2.4% 

3.4% 

2.67 to 2.99 

14.6% 

14.9% 

3.00 to 3.33 

14.6% 

23% 

3.34 to 3.66 

41.5% 

32.2% 

3.67 or higher 

26.8% 

26.4% 


The University uses a +/- grading system. 


Table 2. MFT (Major Field Test Scores) 



N= 41 

N= 87 

135 or lower 

4.9% 

5.7% 

136 to 150 

48.8% 

40.2% 

151 to 165 

36.6% 

39.1% 

166 to 180 

4.9% 

9.2% 

181 or higher 

4.9% 

3.4% 


STUDENT ENGAGEMENT SCALES AND GPA/MFT 

As indicated in Table 3, the data continues to show that the MFT is a better measurement than GPA of 
student engagement as all of the scales of student engagement (Level of Academic Challenge, Active and 
Collaborative Learning, Enriching Educational Experiences, and Enriching Educational Experiences) were positive 
while the GPA relationships were predominately negative. The Integration of Diversity into Coursework scale 
revealed some interesting results. While the partial correlation was by far the strongest in both the MFT and GPA, it 
was positively related to MFT and negatively related to GPA. Additional statistical analysis supported this data as 
the OLS Regression data in Table 4 indicates GPA is negatively related to all of the Integration of Diversity into 
Coursework items. Since many of the questions in this section relate to ‘putting together ideas or concepts from 
different courses when completing assignments’ or ‘the extent that a project or paper required integrating ideas or 
information from various sources’, we believe that while students may struggle with this initially, resulting in lower 
grades for projects or assignments in the course, the overall impact of this practice is positive. So MFT is a better 
and more comprehensive measurement for student engagement. For example, in a Retail Marketing class, students 
typically do well in the marketing section of the company research project but often struggle with the financial 
analysis section of the project. In addition, faculty members have more recently used the introduction, reinforce, 
and apply approach for concepts such as contribution margin leveraging the Bloom’s Taxonomy approach to 
learning. It is a proven strategy that by expecting students to progress from the remembering state to the evaluating 
state is a proven learning approach. 


Table 3. Selected Partial Correlations between GPA/MFT and Student Engagement 



MFT 

GPA 

Level of Academic Challenge 

.077 

-.151 

Active and Collaborative Learning 

.064 

.042 

Enriching Educational Experiences 

.095 

-.070 

Integration of Diversity into Coursework 

.189* 

-.242** 


*, **, *** indicates significance at 10%, 5% and 1%, respectively 
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Table 4. OLS Regression of Integration of Diversity into Coursework Items of Student Engagement on GPA and MFT 



MFT 

GPA 

The number of courses in the 2009-2010 academic year that expected you to put together ideas or 
concepts from different courses when completing assignments or during class discussions 

3.489** 

-.057 

During the 2009-2010 academic year, the extent coursework emphasized synthesizing and organizing 
ideas, information, or experiences into new. more complex interpretations and relationships 

-.900 

-.106 

During the 2009-2010 academic year, the extent that a paper or project that required integrating ideas 
or information from various sources. 

1.646 

-.016 


*, **, *** indicates significance at 10%, 5% and 1%, respectively 


STUDENT ENGAGEMENT ITEMS AND GPA/MFT 

Partial correlations further explored the specific items of engagement associated with internal assessment 
(GPA) and external assessment (MFT). Table 5 reports only those partial correlations that achieved statistical 
significance in either GPA or MFT. Noted again was the trend for GPA to be overall negatively correlated to the 
survey item while MFT was positive in two of the three items. As stated earlier, more students are involved in and 
spend more time doing co-curricular activities. This appears to have an overall negative impact on GPA but a 
positive impact on MFT. Students may be spending more time on these activities and neglecting their day to day 
work in class but since the ETS Business MFT is a one time test that requires very little study time, they perform at 
an acceptable level. The partial correlations of ‘the number of courses in the 2009-2010 academic year that 
expected you to put together ideas or concepts from different courses when completing assignments or dining class 
discussions’ mirrored the OLS regression data as the MFT showed a significant positive correlation to MFT while 
the correlation to GPA, while not significant, was negative. 


Table 5. Partial Correlations between Survey Items and MFT and GPA 



MFT 

GPA 

Co-curricular activities you have done in the past year, (check all that apply) 

.126 

- 319 *** 

The number of courses in the 2009-2010 academic year that expected you to put together ideas 
or concepts from different courses when completing assignments or during class discussions 

.268** 

-.162 

During the 2009-2010 academic year, the extent coursework emphasized synthesizing and 
organizing ideas, information, or experiences into new, more complex interpretations and 
relationships. 

-.069 

-.283** 


*, **, *** indicates significance at 10%, 5% and 1%, respectively 


Finally, gender and rank continue to be significant. The OLS Regression showed males score higher on the 
MLT test than females (-5.754 at the .05 level of significance); however, females tend to have a higher GPA. 
Mirchandani et al. (2001) showed similar results in their study stating “It has been shown that certain subgroups 
within the population, most notably women and minorities, perform less well on standardized tests in general than 
do White males...” (p. 52). Females may be better organized and detail oriented with those traits lending 
themselves to potentially better quality assignments and projects, and consequently higher GPAs. As you would 
expect, seniors scored higher than juniors signifying a maturation and integration of knowledge from their junior to 
their senior year. 

DISCUSSION 

The results of this study are similar to what other researchers have found regarding engagement but we 
took the analysis one step further. By adding the ETS MLT to the study, we are able to link student engagement to 
a desirable business learning outcome (content knowledge in the ETS MLT). The level of significance and small 
sample size are still of some concern but the trends in the data are encouraging. Additional schools have shown 
interest in being a part of our study and we hope to incorporate them during the next academic year. 

The trends in the data did indicate potential steps that institutions could take to increase their ETS MFT 
scores. By linking two externally valid tests, the ETS MFT and NSSE, an institution’s first step could be to 
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encourage specific behaviors in the NSSE that related to higher ETS MFT scores. For example, integrating diversity 
into coursework, specifically, putting together ideas or concepts from different courses when completing 
assignments or during class discussions showed the highest positive partial correlation between a NSSE item and the 
ETS MFT. The second step could be to encourage students to participate in enriching educational experiences such 
as co-currictilar activities, internships, study abroad and holding a position of leadership on campus. On our 
campus, we have over 60 clubs as well as a variety of sororities and fraternities. This participation not only provides 
the students with a social activity but an intellectual one as well. Students in the marketing club often develop 
marketing plans for small business owners. Often included in these plans are cost analyses for various media, 
budgets and return on investment measures. This blends the marketing and finance disciplines that will likely be 
expected of them in a job. We have also witnessed a considerable amount of mentoring in these activities as well as 
increased in-class participation from those students who are active in co-curricular activities. 

There is one note of caution though; Gordon et al. (2008) found "that Enriching Educational Experiences 
have a significant and negative effect on freshman GPA (but no effect on senior GPA)” (p. 26). They believed, and 
we agree, that a plausible explanation could be that too positive an engaging experience detracts from spending time 
on their studies. 

While institutions continue to develop internal and external validation of learning outcomes, our study 
could be useful in determining the type of instrument the institution desires to use as well as where they allocate 
institutional resources regarding assessment. The NSSE and the ETS MFT, at least for business, appear to be two 
very useful tools in understanding and communicating learning outcomes. 

As a noteworthy finding from the analysis, we need to mention that MFT is a better measurement of 
student engagement described in the NSSE rather than GPA. Therefore, it seems that the MFT is a valid external 
measurement of student engagement. In particular, the MFT shows a relative strength in assessing “Integration of 
Diversity into Coursework.” 
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